Tagging of fruit-producing flowers for robotic selective harvesting

ABSTRACT

The locations of flowers on a plant, rather than the locations of agricultural products produced from such flowers, are used to facilitate the performance of harvesting and other agricultural operations in robotic agricultural applications. In some implementations, the identified location of a fruit-producing flower may be used by a robotic device to apply an indicator tag to a flowering plant proximate the flower for later identification when performing various types of directed and automated agricultural operations. In other implementations, the identified location of a fruit-producing flower may be used by a robotic device to anchor a stem of a flowering plant to a predetermined location such that the location of the flower, and of any fruit(s) later produced by such flower, are controlled and/or known when performing subsequent agricultural operations.

BACKGROUND

Robotic and/or automated systems are increasingly being contemplated forthe agricultural industry to improve productivity in planting, growingand/or harvesting crops. One particular application for which suchsystems may be of use is that of selective harvesting, whereagricultural products are harvested from plants without entirelydestroying the plants. Proposed robotic and/or automated approaches toselective harvesting, however, have generally relied on complex machineperception systems and hardware end-effectors that are customized forparticular types of crops, and that are generally unsuitable for usewith other types of crops. As an example, a machine perception systemoptimized to identify a green bell pepper is generally unsuitable foridentifying other types of agricultural products, such as strawberries,as the agricultural products generally share little commonality acrossdifferent types of crops. Moreover, for many crops, identification of anagricultural product for harvest may be complicated by the fact that theagricultural product itself is difficult to distinguish from itssurrounding foliage. A green bell pepper, for example, is often nearlythe same color as the leaves of a pepper plant, so color is generallynot a distinguishing characteristic that can be relied upon to identifya green bell pepper for harvest.

SUMMARY

This specification is directed generally to methods and apparatus thatrely on the locations of flowers on a plant, rather than the locationsof agricultural products produced from such flowers, to facilitate theperformance of harvesting and other agricultural operations. Inparticular, a wide variety of agricultural crops for which selectiveharvesting may be used are classified as angiosperms, or floweringplants, whereby the agricultural products produced by such crops (e.g.,vegetables, nuts, legumes, grains, culinary fruits) are botanicallyconsidered to be fruits that are produced from flowers as part of thenatural reproductive cycle of such crops. Such fruit-producing flowersare believed to be more distinguishable from surrounding foliage thantheir associated agricultural products in many instances, and thus aremore amenable to detection by a machine perception system. Furthermore,such fruit-producing flowers are also believed to share greatercommonality across different types of crops than their associatedagricultural products, reducing the likelihood that customization ortuning is required to adapt a flower detection algorithm for use with aparticular type of crop.

In some implementations, the location of a fruit-producing flower may beidentified and used by a robotic device to apply an indicator tag to aflowering plant proximate the flower. By doing so, the indicator tag maylater be identified to facilitate the performance of various types ofagricultural operations directed to the location of the indicator tag,and as such, directed to the location of the flower and/or any fruit(s)later produced by such flower.

In other implementations, the location of a fruit-producing flower maybe identified and used by a robotic device to anchor a stem of aflowering plant to a predetermined location. By doing so, the locationof the flower, and of any fruit(s) later produced by such flower, arecontrolled and/or known when performing subsequent agriculturaloperations.

Therefore, in some implementations, a method may be provided thatincludes receiving digital image data of at least a portion of aflowering plant, processing the received digital image data to identifya location of a fruit-producing flower on the flowering plant, and priorto production of a fruit by the fruit-producing flower, applying anindicator tag to the flowering plant proximate the identified locationof the fruit-producing flower using a robotic device, the indicator tagbeing configured for detection by a machine perception system toidentify a location of the fruit produced by the fruit-producing flower.

This method and other implementations of technology disclosed herein mayeach optionally include one or more of the following features.

In some implementations, the method further includes locating theindicator tag with the machine perception system, and performing adirected and automated agricultural operation on the flowering plantproximate the indicator tag. In some implementations, applying theindicator tag includes controlling an end-effector of the robotic deviceto apply the indicator tag to a stem of the flowering plant proximatethe fruit-producing flower. In addition, in some implementations,performing the directed and automated agricultural operation includesharvesting the fruit, and in some implementations, harvesting the fruitincludes manipulating the indicator tag to sever the stem and therebyseparate the fruit from the flowering plant.

In some implementations, the indicator tag includes a collapsible loopthat circumscribes the stem, and manipulating the indicator tag includescollapsing the collapsible loop to sever the stem. Further, in someimplementations, the indicator tag includes a cable tie portionincluding a cable that projects through a catch to define thecollapsible loop, controlling the end-effector of the robotic device toapply the indicator tag includes drawing the cable through the catch toa first position, and manipulating the indicator tag to sever the stemincludes drawing the cable further through the catch beyond the firstposition. Further, in some implementations, the cable tie portionincludes at least one cutting element configured to cut the stem whenthe cable is further drawn through the catch.

In some implementations, at least a portion of the indicator tag isreflective, and locating the indicator tag includes detecting areflection of electromagnetic radiation with the machine perceptionsystem. Further, in some implementations, locating the indicator tagfurther includes emitting the electromagnetic radiation with the machineperception system, while in some implementations, the indicator tagincludes a reflector element that is reflective in multiple directions.In some implementations, the indicator tag is configured to hang thereflector element below the stem and below an occluding structure of theflowering plant.

In some implementations, the directed and automated agriculturaloperation includes pollinating the fruit-producing flower, capping thefruit-producing flower to inhibit pollination, transfecting thefruit-producing flower, performing a targeted application of anagricultural fertilizer, pesticide or herbicide to the flowering plant,moving and affixing the stem of the flowering plant to a predeterminedlocation, monitoring growth of the fruit, or determining ripeness of thefruit.

In some implementations, the method further includes storing thelocation of the fruit-producing flower, where locating the indicator tagwith the machine perception system includes retrieving the location ofthe fruit-producing flower to narrow a search space for locating theindicator tag with the machine perception system. Further, in someimplementations, the robotic device is a first robotic device, andwherein locating the indicator tag and performing the directed andautomated agricultural operation on the flowering plant are performed bythe first robotic device or a second robotic device. In someimplementations, receiving the digital image data and processing thereceived digital image data are performed by the robotic device, and insome implementations, the fruit comprises a vegetable agriculturalproduct, a fruit agricultural product, or a nut agricultural product. Inaddition, in some implementations, the indicator tag comprises aVolatile Organic Compound (VOC) sensor.

In some implementations, a method may be provided that includesreceiving digital image data of at least a portion of a flowering plant,processing the received digital image data using a machine perceptionsystem to identify a location of an indicator tag applied to theflowering plant proximate a fruit on the flowering plant and prior toproduction of the fruit by a fruit-producing flower on the floweringplant, and performing a directed and automated agricultural operation onthe flowering plant proximate the indicator tag.

In some implementations, an apparatus may be provided that includes arobotic device including an end-effector configured to apply anindicator tag to a flowering plant, and a control system coupled to therobotic device and configured to receive digital image data of at leasta portion of the flowering plant, process the received digital imagedata to identify a location of a fruit-producing flower on the floweringplant, and prior to production of a fruit by the fruit-producing flower,apply the indicator tag to the flowering plant proximate the identifiedlocation of the fruit-producing flower, where the indicator tag isconfigured for detection by a machine perception system to identify alocation of a fruit produced by the fruit-producing flower on theflowering plant.

In some implementations, the robotic device further includes an imagecapture device configured to collect the digital image data.

In some implementations, an apparatus may be provided that includes arobotic device including an image capture device and an end-effector,the image capture device configured to collect digital image data of atleast a portion of a flowering plant, where the flowering plant includesan indicator tag applied thereto proximate a fruit on the floweringplant and prior to production of the fruit by a fruit-producing floweron the flowering plant, the end-effector configured to perform adirected and automated agricultural operation on the flowering plantproximate the indicator tag, and a control system coupled to the imagecapture device and the end-effector of the robotic device and configuredto process the collected digital image data to identify a location ofthe indicator tag and control the end-effector to perform the directedand automated agricultural operation on the flowering plant proximatethe indicator tag.

In some implementations, a method may be provided that includesreceiving digital image data of at least a portion of a flowering plant,processing the received digital image data to identify a location of afruit-producing flower on the flowering plant, prior to production of afruit by the fruit-producing flower, anchoring a stem of the floweringplant proximate the identified location of the fruit-producing flower toa predetermined location using a robotic device, and after production ofthe fruit by the fruit-producing flower, performing an agriculturaloperation on the flowering plant with the stem of the flowering plantanchored in the predetermined location.

In some implementations, the agricultural operation includes harvestingthe fruit with the stem of the flowering plant anchored in thepredetermined location, while in some implementations, the methodfurther includes processing received digital image data for a pluralityof flowering plants to identify locations of a plurality offruit-producing flowers and using the identified locations to controlthe end-effector of the robotic device to anchor a plurality of stemsproximate the plurality of fruit-producing flowers to a plurality ofpredetermined locations, where performing the agricultural operationincludes performing a cutting operation along a predetermined pathbetween the plurality of predetermined locations to harvest fruit fromthe plurality of flowering plants.

It should be appreciated that all combinations of the foregoing conceptsand additional concepts described in greater detail herein arecontemplated as being part of the subject matter disclosed herein. Forexample, all combinations of claimed subject matter appearing at the endof this disclosure are contemplated as being part of the subject matterdisclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example robotic agricultural system.

FIG. 2 is a top plan view of an example indicator tag capable of beingused with the robotic agricultural system of FIG. 1.

FIG. 3 illustrates the application of the indicator tag of FIG. 2 to thestem of a plant.

FIG. 4 is a top plan view of an another example indicator tag capable ofbeing used with the robotic agricultural system of FIG. 1.

FIG. 5 is a side elevational view of the indicator tag of FIG. 4.

FIG. 6 is a flowchart illustrating an example method of applying anindicator tag.

FIG. 7 is a flowchart illustrating an example method of performing anagricultural operation using an indicator tag.

FIGS. 8-10 are perspective views illustrating an example method ofanchoring a fruit-producing flower and performing an automatedagricultural operation thereon.

FIG. 11 illustrates an example robotic agricultural system.

DETAILED DESCRIPTION

Implementing robotics for selective harvesting of agricultural productsis often complicated by the complexity of the machine perception systemsneeded to identify and locate such agricultural products when harvestinga crop. Different agricultural products can have significantly differentappearances, and in some cases, the agricultural products may bedifficult to distinguish from surrounding foliage. As one example,tomatoes, bell peppers and strawberries all have significantly differentappearances, and detection of the locations of such products forselective harvesting by a robotic system may require different imagerecognition or machine perception algorithms optimized for thoseindividual products. When combined with a need for differentend-effectors customized to harvest particular types of products, thecosts and engineering efforts required to implement robotic selectiveharvesting systems can be prohibitive in some instances.

On the other hand, however, it has been found that differentagricultural products do share some common characteristics. Inparticular, many types of agricultural crops are botanically classifiedas angiosperms, which are otherwise referred to herein as floweringplants. The reproductive cycle of an angiosperm generally begins withthe development of reproductive organs that are commonly known asflowers, and that after fertilization develop or mature into fruit, themeans by which the angiosperm disseminates its seeds. While the term“fruit” is more commonly used in a culinary sense to refer toagricultural products such as apples, bananas, strawberries, etc., froma botanical perspective, a fruit is generally considered to be themature ovary or ovaries of one or more flowers, and as such also refersto a large number of other types of agricultural products, includingmany nuts, legumes, grains and vegetables. As such, referenceshereinafter to a “fruit” will generally refer to fruits in thebotanical, rather than the culinary sense.

Furthermore, it has been found that flowers tend to be much moreself-similar across different types of crops than the agriculturalproducts produced by those crops, e.g., the flower of a strawberry plantlooks more like that of a bell pepper than a strawberry looks like apepper. In addition, due in part to the evolutionary purpose of a flowerin terms of attracting insects for pollination purposes, flowers tend tobe more distinguishable from the surrounding foliage of a crop than dothe fruits. A bell pepper, for example, may be a similar shade of greenas the leaves of a pepper plant, whereas the flower of a pepper plant isgenerally white in color, with pronounced petals having shapes that aresimilar to those of many types of flowers.

Therefore, in various implementations discussed hereinafter, thelocations of fruit-producing flowers, rather than of the fruitsthemselves, may be identified and used for later harvesting of thefruits after maturation of the flowers into fruits and/or for performingother agricultural operations that are directed or targeted towards theflowers and/or the fruits.

In this regard, a fruit-producing flower may be considered to be anyflower-like structure of a flowering plant, or angiosperm, that producesone or more botanical fruits proximate the location of the flower, suchthat the location of the flower is indicative of the location of a fruitproduced by that flower. It will be appreciated that for some plants, aone-to-one correspondence between flowers and fruits may exist, whereasfor other plants, a one-to-many or many-to-one correspondence may existbetween flowers and fruits, and moreover, in some plants, flowers and/orfruits may be clustered together into visually-perceptible groupings. Asone example, the flowers from a grapevine are generally clusteredtogether into inflorescences that produce clusters of grapes, and insuch instances, detection and location of a flower may correspond todetection and location of an inflorescence of flowers. In addition, afruit may correspond to various types of agricultural products,including vegetable agricultural products, fruit agricultural products,legume agricultural products, nut agricultural products, or grainagricultural products, among others.

In some implementations, for example, a machine perception system may beused to detect fruit-producing flowers of flowering plants such that thelocations of such flowers may be identified, and such that indicatortags may be applied to the flowering plants proximate the locations ofsuch flowers by a robotic system. Thereafter, the indicator tags may bedetected by a machine perception system (which may be the same as ordifferent from the machine perception system used to detect theflowers), with the detected locations of the indicator tags used toperform directed and automated agricultural operations on the floweringplants. The agricultural operations may include operations such asharvesting, pollinating, transfecting, capping to inhibit pollination,monitoring, determining ripeness, and targeted applications ofagricultural chemicals such as fertilizer, pesticide, herbicide, etc. Assuch, as used herein, a directed and automated agricultural operationmay be considered to include any agricultural operation that is at leastpartially automated, e.g., performed at least in part using anautonomous or semi-autonomous robotic device, and that is directed ortargeted to a particular location of a flower and/or fruit on a plant.

A machine perception system, in this regard, may include anycomputational system or device incorporating hardware and/or softwareconfigured to process sensory data collected from an environment and toascertain or perceive information about the environment. In theimplementations discussed hereinafter, for example, a machine perceptionsystem includes at least computer vision functionality capable ofprocessing image data of a scene captured by an image capture device todetect or perceive particular objects in the scene. A machine perceptionsystem may be wholly implemented within a robotic device, whollyimplemented within a remote computational device in communication with arobotic device, or collaboratively implemented with some functionalityallocated to both a robotic device and one or more remote computationaldevices.

FIG. 1, for example, illustrates an example robotic agricultural system10 including a robotic device 12 having a robotic arm 14 to which iscoupled an end-effector 16, along with an image capture device 18suitable for capturing digital image data for processing by a machineperception system. A control system 20 may be used to control thevarious components in robotic device 12, and in some implementations, acommunications interface 22 may be provided to enable robotic device 12to communicate with a remote system 24. Part or all of control system 20may be remote from robotic device 12 in some implementations, so acontrol system is not required to be fully implemented within a roboticdevice in some implementations.

Robotic system 10, in this implementation may be used to performagricultural operations on a crop consisting of one or more plants,e.g., plant 30 as illustrated in FIG. 1. Plant 30 is a flowering plant,in this instance a tomato plant, and includes stems or branches 32 andleaves 34 forming the structural support and foliage of the plant. Alsoillustrated in FIG. 1 are flowers 36 and fruits 38, which though bothare illustrated in FIG. 1 for ease of illustration, are generallypresent at different points in time during the development of a plant.In particular, fruits 38 are generally produced by flowers 36 as a partof the natural reproductive cycle of the plant, and the location of aflower 36 on a stem is generally indicative of the location of the fruit38 produced by that flower.

As will become more apparent below, indicator tags may be applied to aplant by a robotic device proximate the locations of flowers on theplant such that the tags may be used to later assist in locating fruitsproduced by those flowers. Tag 40 a, for example, is illustrated in FIG.1 after application proximate a flower 36, whereas tag 40 b isillustrated in FIG. 1 after application proximate a flower that hasthereafter produced a fruit 38, such that tag 40 b has become indicativeof the location of the fruit rather than the flower that existed whenthe tag was originally applied.

As illustrated in FIG. 2, an indicator tag 40 may include structure forsecuring the tag to a plant as well as structure that may be identifiedby a robotic device to thereby identify the location of a flower orfruit with which the tag is associated. Tag 40, for example, isconfigured as a cable tie including a cable tie portion with a cable 42and a catch 44 configured to receive cable 42 and form a collapsibleloop to circumscribe a stem of a plant. Tag 40 also includes anindicator portion having one or more reflector elements 46 coupled tothe cable tie portion by a connector 48.

As illustrated in FIG. 3, in use cable 42 is wrapped around a stem 32and is received within catch 44 to fully circumscribe the stem. Cable 42thereby forms a loop that is collapsible by drawing the cable furtherthrough catch 44, and catch 44 may be configured to resist removal ofcable 42 from the catch, thereby retaining tag 40 on the stem.

In some implementations, the cable tie portion may also be configured toassist with severing stem 32, e.g., to separate and harvest the fruitfrom a plant. In the implementation of FIGS. 2-3, for example, cable 42may be drawn further through catch 44 until the compressional forceapplied by the loop formed by the cable is sufficient to sever the stem.Thus, the cable may be initially drawn through the catch to a positionthat is sufficient to hold the indicator tag on the stem until itbecomes desirable to sever the stem, at which time the stem may besevered by drawing the cable further through the catch.

Moreover, as illustrated in FIGS. 4 and 5, another implementation of anindicator tag 50 may include a cable tie portion including a cable 52and catch 54, and additionally with one or more cutting elements 56formed on the cable 52 to assist with cutting a stem of a plant, whichmay be of use in implementations where stems are woody or otherwisedifficult to sever through compression alone. Various types of cuttingelements, including blades, teeth, barbs, razor wire, serrations, etc.may be used in different implementations.

Other indicator tags may utilize other mechanisms to secure an indicatortag to a plant, including adhesives, looped or wrapped strings or wires,clamps, etc.

Returning to FIGS. 2 and 3, in some implementations, indicator tags maybe specifically constructed to facilitate their identification by amachine perception system, which effectively reduces the complexityrequired to detect the indicator tags. For indicator tag 40, forexample, a passive reflector element 46 may be used to reflect ambientelectromagnetic radiation (e.g., visible light), or to reflect emittedelectromagnetic radiation (e.g., as may be emitted by a focused lightsource on robotic device 12) and thereby distinguish the indicator tagfrom the surrounding foliage of the plant. In one implementation,passive reflector element 46 may be a multi-faceted and mirroredspherical object, similar to a disco ball, to enable the tag to bedetected regardless of the orientation of the tag. In otherimplementations, however, other geometric shapes and reflective surfacesmay be used.

In addition, in some implementations it may be desirable to provideconnectors 48 with sufficient length to position reflector elements 46away from foliage or other occluding structures of a plant. For example,for indicator tags applied to a tree having a dense canopy, it may bedesirable to hang the reflector elements below the canopy to facilitatedetection. Furthermore, the reflector elements may be configured to begrasped by an end-effector of a robotic device such that even when thereflector elements are some distance away from the locations of theflowers or fruits proximate which the indicator tags are applied, thelocations may nonetheless be found by following along the connectorsstarting at the reflector elements.

In other implementations, however, separate reflector elements may notbe used, a reflective surface may be provided on an indicator tag tofacilitate detection. As illustrated in FIGS. 4 and 5, for example, areflective tab 58 may be coupled to catch 54.

It will be appreciated that indicator tags may vary in otherimplementations. For example, rather than passive reflector elements orsurfaces, indicator tags may be provided with active components, e.g.,to emit electromagnetic radiation such as light, to communicateidentification signals and other data (e.g., using active or passiveRFID components), to perform monitoring or sensing operations (e.g.,incorporating Volatile Organic Compound (VOC) sensors to monitor fruitripeness), or to perform other functions in addition to facilitating thelocation of a flower or fruit. In addition, in some implementationsindicator tags may be single use, while in other implementationsindicator tags may be recovered and reused.

It will also be appreciated that an end-effector of a robotic device maybe designed and used to apply indicator tags in a manner appropriate forthe particular type of indicator tag being used, as well as that thesame end-effector or different end-effectors may be designed and used toperform additional operations such as grabbing or grasping an indicatortag and/or fruit, recovering an indicator tag for reuse, severing astem, harvesting a fruit, applying an agricultural chemical such as apesticide, herbicide or fertilizer, capping a flower, transfecting aflower, pollinating a flower, etc.

Further, in some implementations, a source of electromagnetic radiation,as well as the reflective properties of indicator tags, may also betailored to further facilitate detection, e.g., to restrict detection toparticular wavelengths of light, or to use particular wavelengths thatare less likely to be reflected by plant matter and/or to which theplant matter is transparent, such that indicator tags may be locatedmore easily within dense foliage irrespective of leaf occlusion in thevisible spectrum.

FIG. 6 illustrates an example method 100 for applying an indicator tag.Method 100 may be performed, for example, by a robotic device within afield or growing facility, and the operations may be performed by therobotic device in a full autonomous matter in some implementations,while in other implementations the operations may be performed by asemi-autonomous robotic device, e.g., a robotic device mounted onhuman-controlled machinery. In addition, in some implementations all ofthe computational operations may be performed locally by a roboticdevice, while in other implementations a robotic device may be incommunication with a remote computer system that handles a portion ofthe computational operations. In one implementation, for example,digital image processing and recognition to identify flowers in digitalimage data may be performed remotely to leverage high performancecomputational resources in a remote computer system.

In one implementation, it is assumed that a robotic device performsmethod 100, and that the robotic device includes an end-effectorconfigured to apply an indicator tag to a stem of a flowering plant, aswell as an image capture device, e.g., a digital camera, mounted to theend-effector or to a robotic arm to which the end-effector is mounted.Method 100 begins in block 102 by moving the end-effector to a newposition, e.g., by moving the end-effector itself and/or moving theentire robotic device to a new position relative to a flowering plant orto a new flowering plant. Then, in block 104, a digital image iscaptured, and in block 106, the digital image data from the digitalimage is processed to search for one or more flowers in the capturedimage. In block 106, a flower recognition algorithm may be used toattempt to locate any flowers in the captured image. In someimplementations, the flower recognition algorithm may be tuned ortrained on the flowers of a particular species of flowering plant, whilein other implementations, the flower recognition algorithm may be usablewith the flowers of many different species, given that many flowersshare common visual cues such as the presence of petals and typically astark difference in color from surrounding foliage. In addition, in someimplementations, a flower recognition algorithm may also be configuredto ignore flowers not meeting an acceptance criterion, e.g., to avoidapplying tags to flowers that are underdeveloped or that are too closeto other flowers.

Next, in block 108, a FOR loop is performed to process each flowerlocated in the captured digital image. For each such flower, thelocation of the flower in a three-dimensional space may be identified inblock 110, and the end-effector may be moved to the identified locationof the flower in block 112. It will be appreciated that theidentification of the location of the flower and the movement of theend-effector may be an iterative process in some implementations, e.g.,whereby new digital images are captured as the end-effector is movedtoward the flower to assist in guiding the end-effector to the flower.In other implementations, the location of the flower may be determinedand the end-effector may move to the location without further guidance.In addition, a rangefinder or other distance sensor may be used in someimplementations to assist in identifying the location of a flower andguiding an end-effector.

Next, in block 114, once the end-effector has been moved proximate tothe identified location of the flower, the end-effector applies anindicator tag (e.g., any of the tag designs discussed above) to theflowering plant proximate the identified location of the flower, e.g.,around a stem proximate the flower. Control then returns to block 108 torepeat blocks 110-114 for any additional flowers located in the originaldigital image.

Once all flowers have been processed, block 108 then returns control toblock 102 to move the end-effector to a new position and capture a newdigital image, e.g., to capture a digital image of another area of thesame flowering plant or of a different flowering plant in a crop. Method100 may then continue until tags have been applied to all applicableflowers in a crop of flowering plants.

FIG. 7 next illustrates a complementary method 120 for performing anagricultural operation using the previously-applied indicator tags.Method 120 may be performed, for example, by a robotic device within afield or growing facility, and the operations may be performed by therobotic device in a full autonomous matter in some implementations,while in other implementations the operations may be performed by asemi-autonomous robotic device, e.g., a robotic device mounted onhuman-controlled machinery. The robotic device may be the same roboticdevice used to perform method 100, or may be a different robotic deviceconfigured to perform various types of agricultural operations. Inaddition, if multiple agricultural operations are performed over thecourse of a growing season, different robotic devices specialized fordifferent agricultural operations, or including different end-effectorsspecialized for different agricultural operations, may perform method120 at different points in time. As with method 100, in someimplementations all of the computational operations for method 120 maybe performed locally by a robotic device, while in other implementationsa robotic device may be in communication with a remote computer systemthat handles a portion of the computational operations.

Method 120 begins in block 122 by moving the end-effector to a newposition, e.g., by moving the end-effector itself and/or moving theentire robotic device to a new position relative to a flowering plant orto a new flowering plant. Then, in block 124, a digital image iscaptured, and in block 126, the digital image data from the digitalimage is processed to search for one or more indicator tags in thecaptured image. In block 126, a tag recognition algorithm may be used toattempt to locate any indicator tags in the captured image. The tagrecognition algorithm may be tuned or trained to locate the particulartags applied to a flowering plant, and when the tags are configured tobe visually distinct from the flowering plant, generally a less complexalgorithm is required than would generally be the case of a recognitionalgorithm tasked with locating fruits on a flowering plant. In otherimplementations, blocks 124 and 126 may differ based upon the particulartags being detected, e.g., for tag designs where electromagneticradiation is emitted by an end-effector to visually or electronicallyinteract with a tag.

Next, in block 128, a FOR loop is performed to process each tag locatedin the captured digital image. For each such tag, the location of thetag in a three-dimensional space may be identified in block 130, and theend-effector may be moved to the identified location of the tag in block132. It will be appreciated that the identification of the location ofthe tag and the movement of the end-effector may be an iterative processin some implementations, e.g., whereby new digital images are capturedas the end-effector is moved toward the tag to assist in guiding theend-effector to the tag. In other implementations, the location of thetag may be determined and the end-effector may move to the locationwithout further guidance. In addition, a rangefinder or other distancesensor may be used in some implementations to assist in identifying thelocation of a tag and guiding an end-effector.

Next, in block 134, once the end-effector has been moved proximate tothe identified location of the tag, a directed and automatedagricultural operation may be performed at the site of the tag. Anon-exclusive list of potential operations is illustrated by blocks136-146. As illustrated by block 136, for example, a harvestingoperation may be performed to harvest a fruit that has developedproximate the location of the tag. In some implementations, harvestingmay include grabbing or grasping the indicator tag with the end-effectorand manipulating the indicator tag to collapse a loop and sever a stemaround which the indicator tag has been wrapped. In otherimplementations, harvesting may include grabbing or grasping a fruitdirectly, but using the location of the indicator tag to substantiallynarrow the search space for locating the fruit on the flowering plant.

As another example, as illustrated by blocks 138 and 140, someagricultural operations may be directed to the flowers associated withparticular tags. For example, flowers may be pollinated and/or flowersmay be capped to prevent pollination. In some implementations, forexample, it may be desirable to select only a subset of flowers fromwhich to produce fruit to improve the quality of the resulting fruit.

As another example, as illustrated by block 142, it may be desirable tosecure a stem of the flowering plant proximate an indicator tag to apredetermined location. By doing so, further operations may be performedon the flowering plant with the location of a flower and/or fruitproduced therefrom known and locatable without the use of imagerecognition.

As yet another example, as illustrated by blocks 144 and 146, anagricultural operation may be directed to monitoring the growth of aflowering plant or fruit thereof, which may include determining theripeness of a fruit. Image processing may be performed, for example, todetermine a size and/or color of a fruit, and in some implementations,e.g., where a tag includes a VOC sensor, ripeness may be determinedbased upon VOCs emitted by the fruits when ripening.

Other agricultural operations directed to the sites of the flowersand/or fruits of a flowering plant, as identified via applied indicatortags, may be performed in other implementations. For example, afruit-producing flower may be transfected with a genetic sequence (alsoreferred to as a floral dip). Further, multiple agricultural operationsmay be combined in some implementations, e.g., to detect which fruitsare ripe and harvest only the ripe fruits.

Returning to FIG. 7, after performing an agricultural operation, controlreturns to block 128 to repeat blocks 130-134 for any additional tagslocated in the original digital image. Once all tags have beenprocessed, block 128 then returns control to block 122 to move theend-effector to a new position and capture a new digital image, e.g., tocapture a digital image of another area of the same flowering plant orof a different flowering plant in a crop. Method 120 may then continueuntil agricultural operations have been performed for all of the appliedtags in a crop of flowering plants.

One benefit of the aforementioned implementations is the potentialsimplification of the machine perception systems and algorithms neededfor implementation. Rather than having to recognize particular shapesand colors of fruits, fruit-producing flowers, which tend to be easierto detect across a wider variety of plant species, may be detected, andafter tags are applied to a flowering plant, often a relatively simplemachine perception system/algorithm that is customized for theparticular characteristics of an indicator tag may be used to lateridentify the location of a flower and/or fruit produced therefrom toperform future agricultural operations. The reduced challenge ofidentifying the locations of flowers and tags relative to fruits maytherefore lead to greater accuracy and less computational expense andcomplexity. In addition, the need for species-specific customizations ofmachine perception algorithms may be reduced due to the greatersimilarity of flowers across species, as well as the ability to reusethe same tag design for multiple species.

Various modifications may be made in other implementations. For example,it may be desirable in some implementations to store the identifiedlocations of flowers and/or indicator tags (e.g., within athree-dimensional (3D) map) in connection with applying the tags suchthat when an agricultural operation is performed at a later point intime, the identified locations may be retrieved and used to narrow thesearch space for locating indicator tags, thereby reducing the timespent attempting to locate indicator tags in a crop of flowering plants.In addition, in some implementations, no tags may be applied, and theidentified locations of flowers may simply be stored in a robotic deviceor in a remote system to enable the locations to be retrieved at a laterpoint in time and used to narrow the search space when performingsubsequent agricultural operations on the flowers and/or fruit producedtherefrom.

Another implementation of the techniques disclosed herein is illustratedin FIGS. 8-10. In this implementation, rather than applying indicatortags, the identification of a flower on a flowering plant is combinedwith a manipulation of the flowering plant to anchor a stem of theflowering plant to a predetermined location so that the location of thefruit that is ultimately produced from the flower is fixed. In such animplementation, an indicator tag is not required, and futureagricultural operations are performed on the flowering plant based uponthe fixed and known location(s) to which the flowering plant isanchored.

As an example, FIG. 8 illustrates a growing system incorporating a tray150 (e.g., a half-cut PVC pipe including a growing medium such as soil.Seeds may be planted at predetermined spacings along tray 150 to produceflowering plants 152 at predetermined locations along the tray. Asillustrated in FIG. 8, each plant 152 may include one or more flowers154 supported by stems 156. Rather than applying tags to the stems,however, a robotic device may move along the tray anchor stems 156 inslots 158 positioned at predetermined locations along the tray. Forexample, as shown in FIG. 9, the end-effector of a robotic device maymove along tray 150 and identify the locations of flowers in a similarmanner to that described above in connection with FIG. 6. For each suchlocation, the end-effector may grab or grasp the stem 156 and anchor thestem within a corresponding slot 158. One or more closures 160 may beplaced over each slot 158 once a stem is positioned within the slot,such that the stem is effectively anchored at a predetermined locationthat is both fixed and known from the standpoint of performing furtheragricultural operations, and without the need for further imageprocessing or recognition.

As shown in FIG. 10, for example, once fruit 162 is produced by eachflowering plant 152, the locations of the fruit are both fixed andknown, and harvesting of the fruit may be performed in an automatedmanner simply by severing each stem along the edge of the tray. In oneimplementation, for example, a rotating cutting blade 164 driven by anactuator 166 may move along a predetermined path 168 to harvest fruit162 along the length of the tray. In some implementations, such anoperation may be a simple mechanical harvesting operation, e.g., runninga spinning blade along a fixed track, and in some implementations noautomation or robotics may be required to perform the agriculturaloperation. In other implementations, however, such agriculturaloperations may be automated in nature.

FIG. 11 is a block diagram of electronic components in an examplerobotic agricultural system 200. System 200 typically includes at leastone processor 202 which communicates with a number of peripheral devicesvia bus subsystem 204. These peripheral devices may include a storagesubsystem 206, including, for example, a memory subsystem 208 and a filestorage subsystem 210, user interface input devices 212, user interfaceoutput devices 214, and a network interface subsystem 216. The input andoutput devices allow user interaction with system 200. Network interfacesubsystem 216 provides an interface to outside networks and is coupledto corresponding interface devices in other computer systems.

In some implementations, user interface input devices 212 may include akeyboard, pointing devices such as a mouse, trackball, touchpad, orgraphics tablet, a scanner, a touchscreen incorporated into the display,audio input devices such as voice recognition systems, microphones,and/or other types of input devices. In general, use of the term “inputdevice” is intended to include all possible types of devices and ways toinput information into system 200 or onto a communication network.

User interface output devices 214 may include a display subsystem, aprinter, a fax machine, or non-visual displays such as audio outputdevices. The display subsystem may include a cathode ray tube (CRT), aflat-panel device such as a liquid crystal display (LCD), a projectiondevice, or some other mechanism for creating a visible image. Thedisplay subsystem may also provide non-visual display such as via audiooutput devices. In general, use of the term “output device” is intendedto include all possible types of devices and ways to output informationfrom system 200 to the user or to another machine or computer system.

Storage subsystem 206 stores programming and data constructs thatprovide the functionality of some or all of the modules describedherein. For example, the storage subsystem 206 may include the logic toperform selected aspects of the methods of FIG. 6 and/or FIG. 7.

These software modules are generally executed by processor 202 alone orin combination with other processors. Memory subsystem 208 used instorage subsystem 206 can include a number of memories including a mainrandom access memory (RAM) 218 for storage of instructions and dataduring program execution and a read only memory (ROM) 220 in which fixedinstructions are stored. A file storage subsystem 210 can providepersistent storage for program and data files, and may include a harddisk drive, a floppy disk drive along with associated removable media, aCD-ROM drive, an optical drive, or removable media cartridges. Themodules implementing the functionality of certain implementations may bestored by file storage subsystem 210 in the storage subsystem 206, or inother machines accessible by the processor(s) 202.

Bus subsystem 204 provides a mechanism for allowing the variouscomponents and subsystems of system 200 to communicate with each otheras intended. Although bus subsystem 204 is shown schematically as asingle bus, alternative implementations of the bus subsystem may usemultiple busses.

System 200 can be of varying types including a workstation, server,computing cluster, blade server, server farm, or any other dataprocessing system or computing device. Due to the ever-changing natureof computers and networks, the description of system 200 depicted inFIG. 11 is intended only as a specific example for purposes ofillustrating some implementations. Many other configurations of system200 are possible having more or fewer components than the computersystem depicted in FIG. 11.

Furthermore, as implemented within a robotic agricultural application,system 200 may also include additional inputs and/or outputs configuredto operate components of a robotic device. Sensory inputs 222, forexample, may include components such as digital cameras, microphones,and other environmental sensors, as well as additional sensors providingfeedback as to the position or orientation of movable components in arobotic device. Actuator outputs 224, for example, may includecommunication links to actuate motors and other actuators to control themovement of a robotic device. It will be appreciated that othercomponents conventionally found in robotic devices may also beincorporated into system 200 as appropriate. Moreover, system 200 mayimplement, among other functionality, a machine perception system thatreceives inputs from various sensory inputs 222, including an imagecapture device, and processes the inputs to evaluate the surroundingenvironment, e.g., to identify flowers and/or tags in theimplementations discussed herein.

In some embodiments, system 200 may be implemented wholly within arobotic device. In other embodiments, system 200 may be partially orwholly implemented remotely from a robotic device, but in communicationtherewith to control the various operations described herein.

While several implementations have been described and illustratedherein, a variety of other means and/or structures for performing thefunction and/or obtaining the results and/or one or more of theadvantages described herein may be utilized, and each of such variationsand/or modifications is deemed to be within the scope of theimplementations described herein. More generally, all parameters,dimensions, materials, and configurations described herein are meant tobe exemplary and that the actual parameters, dimensions, materials,and/or configurations will depend upon the specific application orapplications for which the teachings is/are used. Those skilled in theart will recognize, or be able to ascertain using no more than routineexperimentation, many equivalents to the specific implementationsdescribed herein. It is, therefore, to be understood that the foregoingimplementations are presented by way of example only and that, withinthe scope of the appended claims and equivalents thereto,implementations may be practiced otherwise than as specificallydescribed and claimed. Implementations of the present disclosure aredirected to each individual feature, system, article, material, kit,and/or method described herein. In addition, any combination of two ormore such features, systems, articles, materials, kits, and/or methods,if such features, systems, articles, materials, kits, and/or methods arenot mutually inconsistent, is included within the scope of the presentdisclosure.

What is claimed is:
 1. A method, comprising: receiving digital imagedata of at least a portion of a flowering plant; processing the receiveddigital image data to identify a location of a fruit-producing flower onthe flowering plant; and prior to production of a fruit by thefruit-producing flower, controlling an end-effector of a robotic deviceto apply an indicator tag to a stem of the flowering plant proximate theidentified location of the fruit-producing flower, the indicator tagbeing configured for detection by a machine perception system toidentify a location of the fruit produced by the fruit-producing flower.2. The method of claim 1, further comprising: locating the indicator tagwith the machine perception system; and performing a directed andautomated agricultural operation on the flowering plant proximate theindicator tag.
 3. The method of claim 2, wherein performing the directedand automated agricultural operation includes harvesting the fruit. 4.The method of claim 3, wherein harvesting the fruit includesmanipulating the indicator tag to sever the stem and thereby separatethe fruit from the flowering plant.
 5. The method of claim 4, whereinthe indicator tag includes a collapsible loop that circumscribes thestem, and wherein manipulating the indicator tag includes collapsing thecollapsible loop to sever the stem.
 6. The method of claim 5, whereinthe indicator tag includes a cable tie portion including a cable thatprojects through a catch to define the collapsible loop, whereincontrolling the end-effector of the robotic device to apply theindicator tag includes drawing the cable through the catch to a firstposition, and wherein manipulating the indicator tag to sever the stemincludes drawing the cable further through the catch beyond the firstposition.
 7. The method of claim 6, wherein the cable tie portionincludes at least one cutting element configured to cut the stem whenthe cable is further drawn through the catch.
 8. The method of claim 2,wherein at least a portion of the indicator tag is reflective, andwherein locating the indicator tag includes detecting a reflection ofelectromagnetic radiation with the machine perception system.
 9. Themethod of claim 8, wherein locating the indicator tag further includesemitting the electromagnetic radiation with the machine perceptionsystem.
 10. The method of claim 8, wherein the indicator tag includes areflector element that is reflective in multiple directions.
 11. Themethod of claim 10, wherein the indicator tag is configured to hang thereflector element below the stem and below an occluding structure of theflowering plant.
 12. The method of claim 2, wherein the directed andautomated agricultural operation includes pollinating thefruit-producing flower, capping the fruit-producing flower to inhibitpollination, transfecting the fruit-producing flower, performing atargeted application of an agricultural fertilizer, pesticide orherbicide to the flowering plant, moving and affixing the stem of theflowering plant to a predetermined location, monitoring growth of thefruit, or determining ripeness of the fruit.
 13. The method of claim 2,further comprising storing the location of the fruit-producing flower,wherein locating the indicator tag with the machine perception systemincludes retrieving the location of the fruit-producing flower to narrowa search space for locating the indicator tag with the machineperception system.
 14. The method of claim 2, wherein the robotic deviceis a first robotic device, and wherein locating the indicator tag andperforming the directed and automated agricultural operation on theflowering plant are performed by the first robotic device or a secondrobotic device.
 15. The method of claim 1, wherein receiving the digitalimage data and processing the received digital image data are performedby the robotic device.
 16. The method of claim 1, wherein the fruitcomprises a vegetable agricultural product, a fruit agriculturalproduct, or a nut agricultural product.
 17. The method of claim 1,wherein the indicator tag comprises a Volatile Organic Compound (VOC)sensor.
 18. An apparatus, comprising: a robotic device including anend-effector configured to apply an indicator tag to a flowering plant;and a control system coupled to the robotic device and configured toreceive digital image data of at least a portion of the flowering plant,process the received digital image data to identify a location of afruit-producing flower on the flowering plant, and prior to productionof a fruit by the fruit-producing flower, control the end-effector toapply the indicator tag to a stem of the flowering plant proximate theidentified location of the fruit-producing flower, wherein the indicatortag is configured for detection by a machine perception system toidentify a location of a fruit produced by the fruit-producing flower onthe flowering plant.
 19. The apparatus of claim 18, wherein the roboticdevice further includes an image capture device configured to collectthe digital image data.
 20. A method, comprising: receiving digitalimage data of at least a portion of a flowering plant; processing thereceived digital image data to identify a location of a fruit-producingflower on the flowering plant; prior to production of a fruit by thefruit-producing flower, anchoring a stem of the flowering plantproximate the identified location of the fruit-producing flower to apredetermined location using a robotic device; and after production ofthe fruit by the fruit-producing flower, performing an agriculturaloperation on the flowering plant with the stem of the flowering plantanchored in the predetermined location.
 21. The method of claim 20,wherein the agricultural operation includes harvesting the fruit withthe stem of the flowering plant anchored in the predetermined location.22. The method of claim 21, further comprising processing receiveddigital image data for a plurality of flowering plants to identifylocations of a plurality of fruit-producing flowers and using theidentified locations to control an end-effector of the robotic device toanchor a plurality of stems proximate the plurality of fruit-producingflowers to a plurality of predetermined locations, wherein performingthe agricultural operation includes performing a cutting operation alonga predetermined path between the plurality of predetermined locations toharvest fruit from the plurality of flowering plants.